Virti-Cue Social Modeling Tutorial Application: Evaluation Jeannette Jackson, Vanita Gupta, Chris Blais, Robin Halbert EDER 679.28 Dr. Michele Jacobsen University of Calgary
Virti-Cue Social Modeling Application: Evaluation The evaluation phase of our product development is an integral part of the design process. It is intended to gather information about users’ experiences when interacting with our tutorial prototypes. The purpose of our current evaluation was to assess the effectiveness of our learning application in meeting the needs of our learners and to "check that users can use the product and that they like it" (Preece, Rogers, & Sharp, 2007, p. 586). To accomplish this, we engaged in usability testing with a representative group of three to five users, for, as noted by Nielsen (2000), “the best results come from testing no more than five users” (para. 1). The decision to engage in parallel design, creating alternate designs at the same time (Nielsen, 2011), was made because of the inconclusive evidence regarding the effectiveness of animations as compared with equivalent static graphics in learning applications (Ayres, Kalyuga, Marcus, & Sweller, 2005; Ayres & Paas, 2007). These multiple design prototypes were considered and tested by our users. The intent of this testing was to determine the effectiveness of an animated tutorial as compared to an equivalent static graphical/textual tutorial in teaching users how to create a new story and add pictures in the Virti-Cue application. All components of the tutorial design were the same in both prototypes, except for the use of a voice over for guidance versus textual instructions, and animation versus still images. The animated tutorial also incorporated introductory and closing music tracks. The research question that we investigated is: Which presentation (dynamic or static) do users prefer in a tutorial application and what are the best features of each? Preece et al. (2007) stated that the purpose of setting usability goals is to establish a “concrete means of assessing various aspects of an interactive product and the user experience” (p. 20). In the evaluation phase, we sought to identify which of the two designs would best meet our users’ needs in terms of effectiveness, efficiency, and memorability. Evaluation Framework: DECIDE Addressing the need to tailor our usability testing methods to our evaluation goals, we selected specific methods that aligned with our goals. As suggested by Preece, Rogers, and Sharp (2002), “goals and questions should guide all evaluation studies” (p. 360) in that they "provide a focus for observation, as the DECIDE framework points out” (p. 360). Hence, we determined the goals of our evaluation based on the DECIDE framework, as outlined below: 1.Determine the goals and Explore the questions – We will observe our users to determine which presentation (dynamic or static) they prefer in a tutorial application and explore related questions in an informal interview, asking users for suggestions concerning the best features of each presentation format. We will also establish a second goal for the next usability evaluation round wherein we would like to measure post tutorial learning gains. At that time usability testing will focus on the application of learning from the final tutorial model to use of the actual Virti-Cue product. 2. Choose the paradigm and techniques – We will conduct parallel prototype testing and record data gathered through observations, informal interviews, and a questionnaire (survey). 3. Identify the practical issues – We will endeavor to have our two prototypes and the online survey ready on the computers and ensure our users get sufficient time to view both tutorials and to complete the online survey. We will determine the roles within our group, including who will observe, record, and question our users. 4. Deal with ethical issues – Since our online survey is voluntary and names are not required, we haven’t prepared a formal consent form, however if this survey was in the field we would ensure that consent forms had been filled out before conducting the survey. 5.Evaluate, analyze, and present the data – Based on the feedback we receive we will evaluate and analyze the results to assist in making informed product design decisions. We will collate informal observation and interview data and examine survey responses in order to identify themes and issues. This will ensure that any design decisions and changes made are based on reliable and valid research. Future pilot testing of both the revised tutorial format and its application for use of the associated Virti-Cue learning application will be conducted. This is a necessary process to complete before launching our product. Evaluation Methodology: Research Foundations As noted above, in our product evaluation we planned to use observation, informal interviews, and a questionnaire, which, according to Preece et al. (2002), "can be used on their own or in conjunction with other methods to clarify or deepen understandings" (p. 398). In the following paragraphs a summary of the research that informed our decisions regarding each of our evaluation methods is presented. Observations In keeping with our evaluation goal, the focus was to observe users to determine which tutorial presentation, dynamic or static, was preferred. As Preece et al. (2002) cautioned, “having a goal, even a very general goal, helps to guide the observation because there is always so much going on” (p. 361). Preece et al. (2002) presented a number of guidelines for observing users that we employed during our usability testing. They recommended noting what is happening; what people are doing and saying; how they are behaving; as well as their tone and body language (Preece et al., 2002, p. 368). In recording our observations, we planned to work as a team, which has several benefits such as allowing us to distribute the workload, focus on different contexts, compare observations, and generate more reliable data (Preece et al., 2002). Having learned from our experiences in observing users during our interaction design, we decided to record observations on a blank sheet, rather than attempting to fill in pre-determined categories. As well, we intended to engage in member checks, (i.e. confirming our interpretations with our users) to ensure that we make good interpretations and improve the ‘validity’ of the evaluation process (Creswell, 2007). As we record our observations, we will endeavor to separate personal opinion from what actually happens, as recommended by Preece et al. (2002). Our observation notes will be reviewed “as soon as possible after each evaluation session to flesh out detail and check ambiguities with other observers” (Preece et al., 2002, p. 369) or our users. Informal Open-Ended Interview As noted by Preece et al. (2002), "interviews can be thought of as a 'conversation with a purpose'" (citing Kahn & Cannell, p. 390), and that was our intent in engaging in an open-ended interview. Given that our evaluation goal was to gain "first impressions about how users react" to our parallel designs, and to "explore users' general attitudes" (Nielsen, 2010, para. 16), an informal, open-ended interview best met our needs and served to provide a rich source of data (Preece et al., 2002, p. 390). Following the advice given by Preece et al. (citing Robson), we planned interview questions that weren't too long; avoided compound questions, jargon, and leading questions; and reflected sensitivity to our own biases by striving for neutrality in our questions. Our interview agenda supported our study goals, as identified in the DECIDE framework. Our intent was to learn about our users' reactions to the parallel designs of the tutorial, therefore our agenda included questions that allowed users to respond openly and freely regarding their impressions of the two formats. Open ended questions such as, "What did you think of this format?" and "How did this format compare to the other?" were to be used to begin the conversation. As Preece et al. (2002) recommended, we will be "prepared to follow new lines of enquiry that contribute to (y)our agenda" (p. 392) as they arise. As noted below in the 'Questionnaire' section, we were aware of "the query effect" (Nielsen, 2010, para. 19) and worked to ensure that the questions asked related directly to our evaluation goals. Questionnaire Preece et al. (2002) offered a number of guidelines for designing questionnaires, many of which we have adopted, including: • avoiding complex multiple questions • making questions clear and specific • beginning with general questions, followed by specific questions • providing clear instructions • using consistent rating scales • aiming for brevity over length to encourage completion (p. 400). In addition, Nielsen (2004) suggested that in order to ensure high response rates and avoid misleading results, you should "keep your surveys short and ensure that your questions are well written and easy to answer" (Summary section). He added, "the highest response rates come when surveys are quick and painless. And the best way to reduce users' time and suffering is to reduce the number of questions" (Nielsen, 2004, para. 3). The most influential of Nielsen's (2004) recommendations in our survey design was his advice regarding “survey bloat” (para. 6): "Please resist the temptation to collect all the information that anybody could ever want. You will end up with no information (or misleading information) instead" (para. 6). Related to this is the "Query Effect", also referred to by Nielsen (2010), who cautioned "whenever you do ask users for their opinions, watch out for the query effect: People can make up an opinion about anything, and they'll do so if asked” (para. 19). The lesson for our group was to be careful in what we asked, making sure it was information that we wanted to have and that mattered in our design. Therefore, questions regarding the overall Virti-Cue product design were not included and our focus remained on questions that served to address our research goal. Evaluation Methodology: Process During testing, we alternated which of the two tutorial design models was accessed first by each of the users. We conducted an informal open-ended interview to gage user reactions after they had viewed both of the tutorial formats, and also administered a follow-up questionnaire to gather related comments, as research supports such initiatives (U. S. Department of Health & Human Services, 2006, p. 190). The questionnaire posed open-ended questions and rating questions regarding the effectiveness of the tutorial in explaining the navigation, ease of use of the tutorial application, and also included a segment to compare the two tutorial formats. Specific questions concerning design choices were included, such as layout, icon clarity, voice-over clarity, and clarity of instructions. During the usability testing, we kept observational notes and tracked users’ progress, as described above. Usability Testing Results and Interpretation Preece et al. (2002) noted, "much of the power of analyzing descriptive data lies in being able to tell a convincing story, illustrated with powerful examples that help to confirm the main points and will be credible" (p. 380). As we analyzed the data from our usability testing, we sought to find those powerful examples that helped to confirm our interpretations and that told a “convincing story” (Preece et al., 2002, p. 380). Employing triangulation in our data gathering methods aided in this, as different methods provided “different perspectives and corroboration of findings across techniques, thus leading to more rigorous and defensible findings” (Preece et al., 2007, p. 293). Following are summaries of the data collected from the three different techniques that we employed. Observations Two members of the Virti-Cue team observed each of our users as they worked their way through our parallel designs. As noted above, having two observers of each situation allowed us to compare our observations and check for accuracy. All members of our team agreed that users had no difficulty in working through the designs, an observation that is supported by the fact that users did not hesitate, or need to stop and ask for help. Users also appeared to be relaxed and comfortable with the testing situation. The digital images that we took of our users provided further evidence of their apparent ease. We therefore concluded that the presentations were straightforward, easy to use, and did not present users with any challenges that they needed to overcome. One user did stop to ask a question regarding the icons that were used in the presentation, and also to question a process that was demonstrated in the tutorial. The user noted that the icon for using a picture (a green checkmark) had been duplicated on two different screens to indicate similar actions, which is a design feature that we will examine and clarify as we move on to future iterations. We felt that the question regarding the process would have been clear had the user been able to interact with the actual Virti-Cue application. Nevertheless, the question may warrant further clarification and specific usability testing to ensure that the processes are presented in a clear and unambiguous manner. We engaged in member checks and users confirmed the accuracy of our observations, indicating that the designs were both easy to navigate and clearly presented the process. A final observation that is worth noting is the memorability of the Virti-Cue jingle. One user suggested that we should have asked questions regarding the memorability of the jingle, as he couldn’t get it out of his head! This was further evidenced as the user continued to interact in the classroom, quietly humming the Virti-Cue song. Informal Open-Ended Interview After users completed each of the tutorial formats, we engaged in an open-ended interview, “a conversation with a purpose” (Preece et al., 2002, citing Kahn & Cannell, p. 390), to gain initial impressions as to how our users reacted to the different formats. We asked open-ended questions such as, “So what did you think of this format? What are your general impressions?” Animated tutorial.There was consistency in users’ initial impressions in that they all felt the animated tutorial was well done. As one user noted, “it’s a well-put-together tutorial.” Users also reacted positively to the length of the tutorial, saying that it was an appropriate length. Other comments focused on the content of the tutorial, as noted above, with regards to the icons used. One user suggested that the ‘save & add picture’ icon/text could be changed to file folder/photo album type icon. Another user suggested that the video tutorial could be presented simultaneously in a split screen, while the user navigated the application. We liked this idea of viewing and doing simultaneously, however we are not certain if it could be implemented since Virti-cue is a downloadable application for use primarily on handheld devices. Navigating on smaller screens while watching the tutorial may not be an easy task to achieve. Nevertheless, we will certainly discuss the possibility of this option in our design team. Static tutorial.After reading the graphical/textual format, users indicated that the directions were clear, however not memorable, as without the application in hand and the ability to apply the instructions, the words became meaningless. We feel that this is an inherent limitation without an operational application. We recognize that in future developments, it will be necessary to have a functional application and that tutorials will need to be evaluated in terms of learning and users’ ability to apply that learning to the functional application. It was also interesting to note one users’ observations: “I don’t think I need the instructions. I think that your interface is laid out in a fairly intuitive manner, so I wouldn’t need to go to a tutorial. The instructions are good, I just don’t think they are necessary.” As this user was experienced with computer applications, including manipulating digital images, this observation is not surprising. As a development team, we agree with this user’s observations - the tutorials are intended for novice users, and may not be needed by the majority, who, as noted by a user, “would probably just use the application.” Survey Monkey Questionnaire Question 1: Format preference - static or dynamic.Most users preferred the animated format (three out of four) as they felt there was too much reading in the text-based tutorial. They found the animated version easier to follow, and appreciated that it attended to visual and auditory learners. Users also said that the animated tutorial “walked through the use of the product.” The fourth user said that he didn’t think either tutorial would be necessary as the “interface is pretty intuitive.” If he had to choose one, he would choose the written one, as “a user would then be able to refer to a paper document and use their computer/smart phone at the same time.” Given that the majority of users preferred the animated version, this is where we will continue to focus our development efforts. The one user who preferred the written tutorial wanted to be able to use the application while reading the instructions. In response to this preference, we feel that by providing downloadable tutorials, the user would be able to use his device while viewing the tutorial on his computer. An alternative to providing only one format is to continue with animated andstatic versions of the tutorial, thus providing our users with a choice. Considering that the content of each format is essentially the same, continuing to create parallel designs would not be a significant factor in terms of cost. We will consider this as an option in our development as we proceed to our next round of usability testing, however will bear in mind advice given by Kevin Cheng (2005), It takes a certain amount of guts to be able to dictate the choices for a user. In my mind, when you’re faced with a dilemma of how something should be done, ‘letting the user choose’ is often a cop-out. Giving people the option to do everything is like designing by focus group; you assume people actually know what they want. (para. 5) As we consider whether to continue with parallel versions of our tutorials, we will do as Cheng advised: “Take a step back and think really long and hard about whether your audience really needs that choice” (para. 11). Question 2: Animated tutorial ease of use.All users agreed that the tutorial was a reasonable length, and that the instructions were clear and easy to follow. It was unanimously ‘strongly agreed’ that the tutorial clearly outlined the task, was appropriately paced, and that the voice was clear and easy to understand. Given these responses, it seems that we are on target with the length and pacing of the tutorial and have also clearly described the task. Question 3: Static tutorial ease of use.All users agreed that the static tutorial provided instructions that were clear and easy to follow and clearly outlined the task. Most users (three out of four) agreed that the tutorial was a reasonable length, whereas one strongly disagreed. This user also commented that without a manipulative (i.e. a functional application), the learning “does not stick”. He added, “I lost track of what was being explained without seeing how things might work as shown in the animation.” As a design team, we understand the importance of having a functional application with which to engage in usability testing and realize that the static tutorial did not allow users to “see how things work.” User comments indicated that they felt the animated tutorial effectively showed the processes and was a preferred format. These comments support our continued focus on the development of animated tutorials, rather than static. Question 4 - Overall tutorial impressions.Users either agreed or strongly agreed that the icons had clear meanings and the visual layout was attractive. One user remarked that the icons were clear to understand however the check mark was confusing; as previously noted, the checkmark is used to indicate two different, yet related processes. As a design team we valued the feedback, for as Norman (2007) said, “feedback provides informative clues about what is happening, clues about what we should do” (p. 138). In considering this feedback, we kept in mind Nelson’s (as cited by Preece et al., 2002) ‘ten minute rule’ that proposes that a novice user should be able to learn how to use a system in under 10 minutes – if not, the system fails. Since the users in our usability testing were able to understand the icons and the tutorial in less than ten minutes, at this time, we have decided not to change the icons, which provide ‘feedback’ to the user. However, with the production of a functional application it may be necessary to revisit the clarity of this particular icon in future usability tests. Implications for Design The results of our usability testing indicated users had a strong preference for the animated tutorial over the static version. Given that the feedback on the animated tutorial was overwhelmingly positive with regards to its length, pacing, clarity, sound, and voice, we feel that we are ‘on target’ with the design choices that we have made to date. We will continue to focus our development efforts on the animated format and will not make any significant changes based on this evaluation study. It is not necessary to combine the ‘best of both formats’, as the animated version was designed to tap many of the positive features of a static presentation. For example, it provides users with the ability to pause, review, or fast forward, thereby controlling the transitory nature of animation (Ayres et al., 2005) and allowing images to be left on the screen for a longer period of time (Ayres & Paas, 2007). Conclusion In summary, we believe we have met our goal in supporting novice users by providing a clear and easy to follow animated tutorial. In the future, we propose further testing of our animated tutorial once we have developed a working prototype on a handheld device. With a prototype for users to navigate through, we will be in a better position to assess how well we achieved the usability goals of our animated tutorial. In these future iterations, we would seek a user scenario that tests efficiency by measuring “the time it takes typical users to complete a task” and memorability by measuring “the number of errors that participants make” (Preece et al., 2007, p. 646). Our hope is to create an efficient, effective, and memorable tutorial format that optimizes the user experience.
References Ayres, P., Kalyuga, S., Marcus, N., & Sweller, J. (2005). The conditions under which instructional animations may be effective. Paper presented at an International Workshop and Mini-conference, Open University of the Netherlands: Heerlen, The Netherlands. Retrieved from www.ou.nl/Docs/Expertise/OTEC/Nieuws/icleps%20conferentie/ Ayres.doc Ayres, P., & Paas, F. (2007). Making instructional animations more effective: A cognitive load approach. Applied Cognitive Psychology, 21, 695-700. doi:10.1002/acp.1343 Cheng, K. (2005). Choosing to give choices. OK/Cancel.com, June 3, 2005.Retrieved from http://okcancel.com/archives/article/2005/06/choosing-to-give-choices.html Creswell, J. (2007). Qualitative inquiry & research. Thousand Oaks, CA: Sage Publications. Nielsen, J. (2010). Interviewing users. Jakob Nielsen's alertbox, July 26, 2010. Retrieved fromhttp://www.useit.com/alertbox/interviews.html Nielsen, J. (2004). Keep online surveys short. Jakob Nielsen's alertbox, February 2, 2004. Retrieved fromhttp://www.useit.com/alertbox/20040202.html Nielsen, J. (2011). Parallel & iterative design + competitive testing = high usability. Jakob Nielsen’s alertbox, January 18, 2011. Retrieved fromhttp://www.useit.com/alertbox/design-diversity-process.html Nielsen, J. (2006). User testing is not entertainment. Jakob Nielsen's alertbox, September 11, 2006. Retrieved fromhttp://www.useit.com/alertbox/user-testing-showbiz.html Nielsen, J. (2000). Why you only need to test with 5 users. Jakob Nielsen’s alertbox, March 19, 2000. Retrieved from http://www.useit.com/alertbox/20000319.html Norman, D. (2007). The design of future things. New York, NY: Basic Books. Preece, J., Rogers, Y., & Sharp, H. (2002). Interaction design: Beyond human-computer interaction (1st ed.). New York, NY: John Wiley & Sons. Preece, J., Rogers, Y., & Sharp, H. (2007). Interaction design: Beyond human-computer interaction (2nd ed.). New York, NY: John Wiley & Sons. U.S. Department of Health & Human Services. (2006). Research-based web design & usability guidelines. Retrieved from http://www.usability.gov/guidelines/index.html
Jeannette Jackson, Vanita Gupta, Chris Blais, Robin Halbert
EDER 679.28
Dr. Michele Jacobsen
University of Calgary
Virti-Cue Social Modeling Application: Evaluation
The evaluation phase of our product development is an integral part of the design process. It is intended to gather information about users’ experiences when interacting with our tutorial prototypes. The purpose of our current evaluation was to assess the effectiveness of our learning application in meeting the needs of our learners and to "check that users can use the product and that they like it" (Preece, Rogers, & Sharp, 2007, p. 586). To accomplish this, we engaged in usability testing with a representative group of three to five users, for, as noted by Nielsen (2000), “the best results come from testing no more than five users” (para. 1).
The decision to engage in parallel design, creating alternate designs at the same time (Nielsen, 2011), was made because of the inconclusive evidence regarding the effectiveness of animations as compared with equivalent static graphics in learning applications (Ayres, Kalyuga, Marcus, & Sweller, 2005; Ayres & Paas, 2007). These multiple design prototypes were considered and tested by our users. The intent of this testing was to determine the effectiveness of an animated tutorial as compared to an equivalent static graphical/textual tutorial in teaching users how to create a new story and add pictures in the Virti-Cue application. All components of the tutorial design were the same in both prototypes, except for the use of a voice over for guidance versus textual instructions, and animation versus still images. The animated tutorial also incorporated introductory and closing music tracks. The research question that we investigated is: Which presentation (dynamic or static) do users prefer in a tutorial application and what are the best features of each?
Preece et al. (2007) stated that the purpose of setting usability goals is to establish a “concrete means of assessing various aspects of an interactive product and the user experience” (p. 20). In the evaluation phase, we sought to identify which of the two designs would best meet our users’ needs in terms of effectiveness, efficiency, and memorability.
Evaluation Framework: DECIDE
Addressing the need to tailor our usability testing methods to our evaluation goals, we selected specific methods that aligned with our goals. As suggested by Preece, Rogers, and Sharp (2002), “goals and questions should guide all evaluation studies” (p. 360) in that they "provide a focus for observation, as the DECIDE framework points out” (p. 360). Hence, we determined the goals of our evaluation based on the DECIDE framework, as outlined below:
1. Determine the goals and Explore the questions – We will observe our users to determine which presentation (dynamic or static) they prefer in a tutorial application and explore related questions in an informal interview, asking users for suggestions concerning the best features of each presentation format. We will also establish a second goal for the next usability evaluation round wherein we would like to measure post tutorial learning gains. At that time usability testing will focus on the application of learning from the final tutorial model to use of the actual Virti-Cue product.
2. Choose the paradigm and techniques – We will conduct parallel prototype testing and record data gathered through observations, informal interviews, and a questionnaire (survey).
3. Identify the practical issues – We will endeavor to have our two prototypes and the online survey ready on the computers and ensure our users get sufficient time to view both tutorials and to complete the online survey. We will determine the roles within our group, including who will observe, record, and question our users.
4. Deal with ethical issues – Since our online survey is voluntary and names are not required, we haven’t prepared a formal consent form, however if this survey was in the field we would ensure that consent forms had been filled out before conducting the survey.
5. Evaluate, analyze, and present the data – Based on the feedback we receive we will evaluate and analyze the results to assist in making informed product design decisions. We will collate informal observation and interview data and examine survey responses in order to identify themes and issues. This will ensure that any design decisions and changes made are based on reliable and valid research. Future pilot testing of both the revised tutorial format and its application for use of the associated Virti-Cue learning application will be conducted. This is a necessary process to complete before launching our product.
Evaluation Methodology: Research Foundations
As noted above, in our product evaluation we planned to use observation, informal interviews, and a questionnaire, which, according to Preece et al. (2002), "can be used on their own or in conjunction with other methods to clarify or deepen understandings" (p. 398). In the following paragraphs a summary of the research that informed our decisions regarding each of our evaluation methods is presented.
Observations
In keeping with our evaluation goal, the focus was to observe users to determine which tutorial presentation, dynamic or static, was preferred. As Preece et al. (2002) cautioned, “having a goal, even a very general goal, helps to guide the observation because there is always so much going on” (p. 361). Preece et al. (2002) presented a number of guidelines for observing users that we employed during our usability testing. They recommended noting what is happening; what people are doing and saying; how they are behaving; as well as their tone and body language (Preece et al., 2002, p. 368). In recording our observations, we planned to work as a team, which has several benefits such as allowing us to distribute the workload, focus on different contexts, compare observations, and generate more reliable data (Preece et al., 2002). Having learned from our experiences in observing users during our interaction design, we decided to record observations on a blank sheet, rather than attempting to fill in pre-determined categories. As well, we intended to engage in member checks, (i.e. confirming our interpretations with our users) to ensure that we make good interpretations and improve the ‘validity’ of the evaluation process (Creswell, 2007). As we record our observations, we will endeavor to separate personal opinion from what actually happens, as recommended by Preece et al. (2002). Our observation notes will be reviewed “as soon as possible after each evaluation session to flesh out detail and check ambiguities with other observers” (Preece et al., 2002, p. 369) or our users.
Informal Open-Ended Interview
As noted by Preece et al. (2002), "interviews can be thought of as a 'conversation with a purpose'" (citing Kahn & Cannell, p. 390), and that was our intent in engaging in an open-ended interview. Given that our evaluation goal was to gain "first impressions about how users react" to our parallel designs, and to "explore users' general attitudes" (Nielsen, 2010, para. 16), an informal, open-ended interview best met our needs and served to provide a rich source of data (Preece et al., 2002, p. 390). Following the advice given by Preece et al. (citing Robson), we planned interview questions that weren't too long; avoided compound questions, jargon, and leading questions; and reflected sensitivity to our own biases by striving for neutrality in our questions. Our interview agenda supported our study goals, as identified in the DECIDE framework. Our intent was to learn about our users' reactions to the parallel designs of the tutorial, therefore our agenda included questions that allowed users to respond openly and freely regarding their impressions of the two formats. Open ended questions such as, "What did you think of this format?" and "How did this format compare to the other?" were to be used to begin the conversation. As Preece et al. (2002) recommended, we will be "prepared to follow new lines of enquiry that contribute to (y)our agenda" (p. 392) as they arise. As noted below in the 'Questionnaire' section, we were aware of "the query effect" (Nielsen, 2010, para. 19) and worked to ensure that the questions asked related directly to our evaluation goals.
Questionnaire
Preece et al. (2002) offered a number of guidelines for designing questionnaires, many of which we have adopted, including:
• avoiding complex multiple questions
• making questions clear and specific
• beginning with general questions, followed by specific questions
• providing clear instructions
• using consistent rating scales
• aiming for brevity over length to encourage completion (p. 400).
In addition, Nielsen (2004) suggested that in order to ensure high response rates and avoid misleading results, you should "keep your surveys short and ensure that your questions are well written and easy to answer" (Summary section). He added, "the highest response rates come when surveys are quick and painless. And the best way to reduce users' time and suffering is to reduce the number of questions" (Nielsen, 2004, para. 3). The most influential of Nielsen's (2004) recommendations in our survey design was his advice regarding “survey bloat” (para. 6): "Please resist the temptation to collect all the information that anybody could ever want. You will end up with no information (or misleading information) instead" (para. 6). Related to this is the "Query Effect", also referred to by Nielsen (2010), who cautioned "whenever you do ask users for their opinions, watch out for the query effect: People can make up an opinion about anything, and they'll do so if asked” (para. 19). The lesson for our group was to be careful in what we asked, making sure it was information that we wanted to have and that mattered in our design. Therefore, questions regarding the overall Virti-Cue product design were not included and our focus remained on questions that served to address our research goal.
Evaluation Methodology: Process
During testing, we alternated which of the two tutorial design models was accessed first by each of the users. We conducted an informal open-ended interview to gage user reactions after they had viewed both of the tutorial formats, and also administered a follow-up questionnaire to gather related comments, as research supports such initiatives (U. S. Department of Health & Human Services, 2006, p. 190). The questionnaire posed open-ended questions and rating questions regarding the effectiveness of the tutorial in explaining the navigation, ease of use of the tutorial application, and also included a segment to compare the two tutorial formats. Specific questions concerning design choices were included, such as layout, icon clarity, voice-over clarity, and clarity of instructions. During the usability testing, we kept observational notes and tracked users’ progress, as described above.
Usability Testing Results and Interpretation
Preece et al. (2002) noted, "much of the power of analyzing descriptive data lies in being able to tell a convincing story, illustrated with powerful examples that help to confirm the main points and will be credible" (p. 380). As we analyzed the data from our usability testing, we sought to find those powerful examples that helped to confirm our interpretations and that told a “convincing story” (Preece et al., 2002, p. 380). Employing triangulation in our data gathering methods aided in this, as different methods provided “different perspectives and corroboration of findings across techniques, thus leading to more rigorous and defensible findings” (Preece et al., 2007, p. 293). Following are summaries of the data collected from the three different techniques that we employed.
Observations
Two members of the Virti-Cue team observed each of our users as they worked their way through our parallel designs. As noted above, having two observers of each situation allowed us to compare our observations and check for accuracy. All members of our team agreed that users had no difficulty in working through the designs, an observation that is supported by the fact that users did not hesitate, or need to stop and ask for help. Users also appeared to be relaxed and comfortable with the testing situation. The digital images that we took of our users provided further evidence of their apparent ease. We therefore concluded that the presentations were straightforward, easy to use, and did not present users with any challenges that they needed to overcome. One user did stop to ask a question regarding the icons that were used in the presentation, and also to question a process that was demonstrated in the tutorial. The user noted that the icon for using a picture (a green checkmark) had been duplicated on two different screens to indicate similar actions, which is a design feature that we will examine and clarify as we move on to future iterations. We felt that the question regarding the process would have been clear had the user been able to interact with the actual Virti-Cue application. Nevertheless, the question may warrant further clarification and specific usability testing to ensure that the processes are presented in a clear and unambiguous manner.
We engaged in member checks and users confirmed the accuracy of our observations, indicating that the designs were both easy to navigate and clearly presented the process. A final observation that is worth noting is the memorability of the Virti-Cue jingle. One user suggested that we should have asked questions regarding the memorability of the jingle, as he couldn’t get it out of his head! This was further evidenced as the user continued to interact in the classroom, quietly humming the Virti-Cue song.
Informal Open-Ended Interview
After users completed each of the tutorial formats, we engaged in an open-ended interview, “a conversation with a purpose” (Preece et al., 2002, citing Kahn & Cannell, p. 390), to gain initial impressions as to how our users reacted to the different formats. We asked open-ended questions such as, “So what did you think of this format? What are your general impressions?”
Animated tutorial. There was consistency in users’ initial impressions in that they all felt the animated tutorial was well done. As one user noted, “it’s a well-put-together tutorial.” Users also reacted positively to the length of the tutorial, saying that it was an appropriate length. Other comments focused on the content of the tutorial, as noted above, with regards to the icons used. One user suggested that the ‘save & add picture’ icon/text could be changed to file folder/photo album type icon. Another user suggested that the video tutorial could be presented simultaneously in a split screen, while the user navigated the application. We liked this idea of viewing and doing simultaneously, however we are not certain if it could be implemented since Virti-cue is a downloadable application for use primarily on handheld devices. Navigating on smaller screens while watching the tutorial may not be an easy task to achieve. Nevertheless, we will certainly discuss the possibility of this option in our design team.
Static tutorial. After reading the graphical/textual format, users indicated that the directions were clear, however not memorable, as without the application in hand and the ability to apply the instructions, the words became meaningless. We feel that this is an inherent limitation without an operational application. We recognize that in future developments, it will be necessary to have a functional application and that tutorials will need to be evaluated in terms of learning and users’ ability to apply that learning to the functional application.
It was also interesting to note one users’ observations: “I don’t think I need the instructions. I think that your interface is laid out in a fairly intuitive manner, so I wouldn’t need to go to a tutorial. The instructions are good, I just don’t think they are necessary.” As this user was experienced with computer applications, including manipulating digital images, this observation is not surprising. As a development team, we agree with this user’s observations - the tutorials are intended for novice users, and may not be needed by the majority, who, as noted by a user, “would probably just use the application.”
Survey Monkey Questionnaire
Question 1: Format preference - static or dynamic. Most users preferred the animated format (three out of four) as they felt there was too much reading in the text-based tutorial. They found the animated version easier to follow, and appreciated that it attended to visual and auditory learners. Users also said that the animated tutorial “walked through the use of the product.” The fourth user said that he didn’t think either tutorial would be necessary as the “interface is pretty intuitive.” If he had to choose one, he would choose the written one, as “a user would then be able to refer to a paper document and use their computer/smart phone at the same time.”
Given that the majority of users preferred the animated version, this is where we will continue to focus our development efforts. The one user who preferred the written tutorial wanted to be able to use the application while reading the instructions. In response to this preference, we feel that by providing downloadable tutorials, the user would be able to use his device while viewing the tutorial on his computer.
An alternative to providing only one format is to continue with animated and static versions of the tutorial, thus providing our users with a choice. Considering that the content of each format is essentially the same, continuing to create parallel designs would not be a significant factor in terms of cost. We will consider this as an option in our development as we proceed to our next round of usability testing, however will bear in mind advice given by Kevin Cheng (2005),
It takes a certain amount of guts to be able to dictate the choices for a user. In my mind, when you’re faced with a dilemma of how something should be done, ‘letting the user choose’ is often a cop-out. Giving people the option to do everything is like designing by focus group; you assume people actually know what they want. (para. 5)
As we consider whether to continue with parallel versions of our tutorials, we will do as Cheng advised: “Take a step back and think really long and hard about whether your audience really needs that choice” (para. 11).
Question 2: Animated tutorial ease of use. All users agreed that the tutorial was a reasonable length, and that the instructions were clear and easy to follow. It was unanimously ‘strongly agreed’ that the tutorial clearly outlined the task, was appropriately paced, and that the voice was clear and easy to understand. Given these responses, it seems that we are on target with the length and pacing of the tutorial and have also clearly described the task.
Question 3: Static tutorial ease of use. All users agreed that the static tutorial provided instructions that were clear and easy to follow and clearly outlined the task. Most users (three out of four) agreed that the tutorial was a reasonable length, whereas one strongly disagreed. This user also commented that without a manipulative (i.e. a functional application), the learning “does not stick”. He added, “I lost track of what was being explained without seeing how things might work as shown in the animation.”
As a design team, we understand the importance of having a functional application with which to engage in usability testing and realize that the static tutorial did not allow users to “see how things work.” User comments indicated that they felt the animated tutorial effectively showed the processes and was a preferred format. These comments support our continued focus on the development of animated tutorials, rather than static.
Question 4 - Overall tutorial impressions. Users either agreed or strongly agreed that the icons had clear meanings and the visual layout was attractive. One user remarked that the icons were clear to understand however the check mark was confusing; as previously noted, the checkmark is used to indicate two different, yet related processes. As a design team we valued the feedback, for as Norman (2007) said, “feedback provides informative clues about what is happening, clues about what we should do” (p. 138). In considering this feedback, we kept in mind Nelson’s (as cited by Preece et al., 2002) ‘ten minute rule’ that proposes that a novice user should be able to learn how to use a system in under 10 minutes – if not, the system fails. Since the users in our usability testing were able to understand the icons and the tutorial in less than ten minutes, at this time, we have decided not to change the icons, which provide ‘feedback’ to the user. However, with the production of a functional application it may be necessary to revisit the clarity of this particular icon in future usability tests.
Implications for Design
The results of our usability testing indicated users had a strong preference for the animated tutorial over the static version. Given that the feedback on the animated tutorial was overwhelmingly positive with regards to its length, pacing, clarity, sound, and voice, we feel that we are ‘on target’ with the design choices that we have made to date. We will continue to focus our development efforts on the animated format and will not make any significant changes based on this evaluation study. It is not necessary to combine the ‘best of both formats’, as the animated version was designed to tap many of the positive features of a static presentation. For example, it provides users with the ability to pause, review, or fast forward, thereby controlling the transitory nature of animation (Ayres et al., 2005) and allowing images to be left on the screen for a longer period of time (Ayres & Paas, 2007).
Conclusion
In summary, we believe we have met our goal in supporting novice users by providing a clear and easy to follow animated tutorial. In the future, we propose further testing of our animated tutorial once we have developed a working prototype on a handheld device. With a prototype for users to navigate through, we will be in a better position to assess how well we achieved the usability goals of our animated tutorial. In these future iterations, we would seek a user scenario that tests efficiency by measuring “the time it takes typical users to complete a task” and memorability by measuring “the number of errors that participants make” (Preece et al., 2007, p. 646). Our hope is to create an efficient, effective, and memorable tutorial format that optimizes the user experience.
References
Ayres, P., Kalyuga, S., Marcus, N., & Sweller, J. (2005). The conditions under which instructional animations may be effective. Paper presented at an International Workshop and Mini-conference, Open University of the Netherlands: Heerlen, The Netherlands. Retrieved from www.ou.nl/Docs/Expertise/OTEC/Nieuws/icleps%20conferentie/ Ayres.doc
Ayres, P., & Paas, F. (2007). Making instructional animations more effective: A cognitive load approach. Applied Cognitive Psychology, 21, 695-700. doi:10.1002/acp.1343
Cheng, K. (2005). Choosing to give choices. OK/Cancel.com, June 3, 2005. Retrieved from http://okcancel.com/archives/article/2005/06/choosing-to-give-choices.html
Creswell, J. (2007). Qualitative inquiry & research. Thousand Oaks, CA: Sage Publications.
Nielsen, J. (2010). Interviewing users. Jakob Nielsen's alertbox, July 26, 2010. Retrieved from http://www.useit.com/alertbox/interviews.html
Nielsen, J. (2004). Keep online surveys short. Jakob Nielsen's alertbox, February 2, 2004. Retrieved from http://www.useit.com/alertbox/20040202.html
Nielsen, J. (2011). Parallel & iterative design + competitive testing = high usability. Jakob Nielsen’s alertbox, January 18, 2011. Retrieved from http://www.useit.com/alertbox/design-diversity-process.html
Nielsen, J. (2006). User testing is not entertainment. Jakob Nielsen's alertbox, September 11, 2006. Retrieved from http://www.useit.com/alertbox/user-testing-showbiz.html
Nielsen, J. (2000). Why you only need to test with 5 users. Jakob Nielsen’s alertbox, March 19, 2000. Retrieved from http://www.useit.com/alertbox/20000319.html
Norman, D. (2007). The design of future things. New York, NY: Basic Books.
Preece, J., Rogers, Y., & Sharp, H. (2002). Interaction design: Beyond human-computer interaction (1st ed.). New York, NY: John Wiley & Sons.
Preece, J., Rogers, Y., & Sharp, H. (2007). Interaction design: Beyond human-computer interaction (2nd ed.). New York, NY: John Wiley & Sons.
U.S. Department of Health & Human Services. (2006). Research-based web design & usability guidelines. Retrieved from http://www.usability.gov/guidelines/index.html